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Abstract 

We study distributed algorithms for solving global optimization problems in 
which the objective function is the sum of local objective functions of agents and 
the constraint set is given by the intersection of local constraint sets of agents. 
We assume that each agent knows only his own local objective function and con- 
straint set, and exchanges information with the other agents over a randomly 
varying network topology to update his information state. We assume a state- 
dependent communication model over this topology: communication is Markovian 
with respect to the states of the agents and the probability with which the links 
are available depends on the states of the agents. 

In this paper, we study a projected multi-agent subgradient algorithm under 
state-dependent communication. The algorithm involves each agent performing a 
local averaging to combine his estimate with the other agents' estimates, taking a 
subgradient step along his local objective function, and projecting the estimates 
on his local constraint set. The state-dependence of the communication introduces 
significant challenges and couples the study of information exchange with the 
analysis of subgradient steps and projection errors. We first show that the multi- 
agent subgradient algorithm when used with a constant stepsize may result in 
the agent estimates to diverge with probability one. Under some assumptions on 
the stepsize sequence, we provide convergence rate bounds on a "disagreement 
metric" between the agent estimates. Our bounds are time-nonhomogeneous in 
the sense that they depend on the initial starting time. Despite this, we show that 
agent estimates reach an almost sure consensus and converge to the same optimal 
solution of the global optimization problem with probability one under different 
assumptions on the local constraint sets and the stepsize sequence. 
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1 Introduction 



Due to computation, communication, and energy constraints, several control and sensing 
tasks are currently performed collectively by a large network of autonomous agents. Ap- 
plications are vast including a set of sensors collecting and processing information about 
a time- varying spatial field (e.g., to monitor temperature levels or chemical concentra- 
tions), a collection of mobile robots performing dynamic tasks spread over a region, 
mobile relays providing wireless communication services, and a set of humans aggregat- 
ing information and forming beliefs about social issues over a network. These problems 
motivated a large literature focusing on design of optimization, control, and learning 
methods that can operate using local information and are robust to dynamic changes 
in the network topology. The standard approach in this literature involves considering 
"consensus-based" schemes, in which agents exchange their local estimates (or states) 
with their neighbors with the goal of aggregating information over an exogenous (fixed 
or time-varying) network topology. In many of the applications, however, the relevant 
network topology is configured endogenously as a function of the agent states, for ex- 
ample, the communication network varies as the location of mobile robots changes in 
response to the objective they are trying to achieve. A related set of problems arises 
when the current information of decentralized agents influences their potential commu- 
nication pattern, which is relevant in the context of sensing applications and in social 
settings where disagreement between the agents would put constraints on the amount 
of communication among them. 

In this paper, we propose a general framework for design and analysis of distributed 
multi-agent optimization algorithms with state dependent communication. Our model 
involves a network of m agents, each endowed with a local objective function /j : — )■ 
M and a local constraint Xi C M" that are private information, i.e., each agent only 
knows its own objective and constraint. The goal is to design distributed algorithms for 
solving a global constrained optimization problem for optimizing an objective function, 
which is the sum of the local agent objective functions, subject to a constraint set 
given by the intersection of the local constraint sets of the agents. These algorithms 
involve each agent maintaining an estimate (or state) about the optimal solution of the 
global optimization problem and update this estimate based on local information and 
processing, and information obtained from the other agents. 

We assume that agents communicate over a network with randomly varying topology. 
Our random network topology model has two novel features: First, we assume that the 
communication at each time instant k, (represented by a communication matrix A{k) 
with positive entries denoting the availability of the links between agents) is Markovian 
on the states of the agents. This captures the time correlation of communication pat- 
terns among the agents]^ The second, more significant feature of our model is that the 
probability of communication between any two agents at any time is a function of the 
agents' states, i.e., the closer the states of the two agents, the more likely they are to 

"'^Note that our model can easily be extended to model Markovian dependence on other stochastic 
processes, such as channel states, to capture time correlation due to global network effects. We do not 
do so here for notational simplicity. 
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communicate. As outlined above, this feature is essential in problems where the state 
represents the position of the agents in sensing and coordination applications or the 
beliefs of agents in social settings. 

For this problem, we study a projected multi- agent suhgradient algorithm, which in- 
volves each agent performing a local averaging to combine his estimate with the other 
agents' estimates he has access to, taking a subgradient step along his local objective 
function, and projecting the estimates on his local constraint set. We represent these it- 
erations as stochastic linear time-varying update rules that involve the agent estimates, 
subgradients and projection errors explicitly. With this representation, the evolution 
of the estimates can be written in terms of stochastic transition matrices ^{k, s) for 
> s > 0, which are products of communication matrices A{t) over a window from time 
s to time k. The transition matrices $(/c,s) represent aggregation of information over 
the network as a result of local exchanges among the agents, i.e., in the long run, it is 
desirable for the transition matrices to converge to a uniform distribution, hence aligning 
the estimates of the agents with uniform weights given to each (ensuring that informa- 
tion of each agent affects the resulting estimate uniformly). As a result, the analysis 
of our algorithm involves studying convergence properties of transition matrices, under- 
standing the limiting behavior of projection errors, and finally studying the algorithm 
as an "approximate subgradient algorithm" with bounds on errors due to averaging and 
projections. 

In view of the dependence of information exchange on the agent estimates, it is not 
possible to decouple the effect of stepsizes and subgradients from the convergence of the 
transition matrices. We illustrate this point by first presenting an example in which the 
projected multi-agent subgradient algorithm is used with a constant stepsize a{k) = a 
for all k > 0. We show that in this gent estimates and the corresponding global 

objective function values may diverge with probability one for any constant value of the 
stepsize. This is in contrast to the analysis of multi-agent algorithms over exogenously 
varying network topologies where it is possible to provide error bounds on the difference 
between the limiting objective function values of agent estimates and the optimal value 
as a function of the constant stepsize a (see [15]). 

We next adopt an assumption on the stepsize sequence {a{k)} (see Assumption [s]), 
which ensures that a{k) decreases to zero sufficiently fast, while satisfying ^fclo'^(^) ~ 
oo and YlkLo^^W < conditions. Under this assumption, we provide a bound on the 
expected value of the disagreement metric, defined as the difference maxjj- |[$(fc, s)]ij — 
^ I . Our analysis is novel and involves constructing and bounding (uniformly) the prob- 
ability of a hierarchy of events, the length of which is specifically tailored to grow faster 
than the stepsize sequence, to ensure propagation of information across the network 
before the states drift away too much from each other. In contrast to exogenous com- 
munication models, our bound is time-nonhomogeneous, i.e., it depends on the initial 
starting time s as well as the time difference {k — s). We also consider the case where 
we have the assumption that the agent constraint sets Xj's are compact, in which case 
we can provide a bound on the disagreement metric without any assumptions on the 
stepsize sequence. 

Our next set of results study the convergence behavior of agent estimates under 
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different conditions on the constraint sets and stepsize sequences. We first study the 
case when the local constraint sets of agents are the same, i.e., for all i, Xi = X 
for some nonempty closed convex set. In this case, using the time-nonhomogeneous 
contraction provided on the disagreement metric, we show that agent estimates reach 
almost sure consensus under the assumption that stepsize sequence {a{k)} converges 
to sufficiently fast (as stated in Assumption |5]). Moreover, we show that under the 
additional assumption ^^o'^(^) ~ estimates converge to the same optimal 

point of the global optimization problem with probability one. We then consider the 
case when the constraint sets of the agents Xi are different convex compact sets and 
present convergence results both in terms of almost sure consensus of agent estimates 
and almost sure convergence of the agent estimates to an optimal solution under weaker 
assumptions on the stepsize sequence. 

Our paper contributes to the growing literature on multi-agent optimization, control, 
and learning in large-scale networked systems. Most work in this area builds on the 
seminal work by Tsitsiklis |26] and Bertsekas and Tsitsiklis [3] (see also Tsitsiklis et 
al. [2ZI), which developed a general framework for parallel and distributed computation 
among different processors. Our work is related to different strands of literature in this 
area. 

One strand focuses on reaching consensus on a particular scalar value or computing 
exact averages of the initial values of the agents, as natural models of cooperative be- 
havior in networked-systems (for deterministic models, see [2H], [Il],[2n], 0, [21], and 
|22j : for randomized models, where the randomness may be due to the choice of the 
randomized communication protocol or due to the unpredictability in the environment 
that the information exchange takes place, see [8], [13], [29], [21], [25], and fTO]) An- 
other recent literature studies optimization of more general objective functions using 
subgradient algorithms and consensus-type mechanisms (see [IS], [IT], [IH], [IS], [TB] , 
[23], [SO])- Of particular relevance to our work are the papers [U] and [IH]. In [U], the 
authors studied a multi-agent unconstrained optimization algorithm over a random net- 
work topology which varies independently over time and established convergence results 
for diminishing and constant stepsize rules. The paper [19j considered multi-agent opti- 
mization algorithms under deterministic assumptions on the network topology and with 
constraints on agent estimates. It provided a convergence analysis for the case when 
the agent constraint sets are the same. A related, but somewhat distinct literature, 
uses consensus-type schemes to model opinion dynamics over social networks (see [I2] . 
[TT] . [I], [6], [5]). Among these papers, most related to our work are [6] and [5], which 
studied dynamics with opinion-dependent communication, but without any optimization 
objective. 

The rest of the paper is organized as follows: in Section[2| we present the optimization 
problem, the projected subgradient algorithm and the communication model. We also 
show a counterexample that demonstrates that there are problem instances where this 
algorithm, with a constant stepsize, does not solve the desired problem. In Section [3} 
we introduce and bound the disagreement metric p, which determines the spread of 
information in the network. In Section |4| we build on the earlier bounds to show the 
convergence of the projected subgradient methods. Section [5] concludes. 
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Notation and Basic Relations: 

A vector is viewed as a column vector, unless clearly stated otherwise. We denote 
by Xi or [x]i the i-th component of a vector x. When Xj > for all components i of a 
vector X, we write x > 0. For a matrix A, we write Aij or [A]ij to denote the matrix 
entry in the i-th row and j-th column. We denote the nonnegative orthant by M", i.e., 
M" = {x E I x > 0}. We write x' to denote the transpose of a vector x. The scalar 
product of two vectors G M" is denoted by x'y. We use ||x|| to denote the standard 
Euclidean norm, ||x|| = \/x'x. 

A vector a G M"^ is said to be a stochastic vectorwhen its components a,, i = 1, . . . , m, 
are nonnegative and their sum is equal to 1, i.e., Yl^i c^i = 1- A square m x m matrix 
A is said to be a stochastic matrix when each row of A is a stochastic vector. A square 
m X m matrix A is said to be a doubly stochastic matrix when both A and A' are 
stochastic matrices. 

For a function F : M" — t- (— oo, oo], we denote the domain of F by dom(F), where 

dom(F) = {x G I F{x) < oo}. 

We use the notion of a subgradient of a convex function F{x) at a given vector x G 
dom(F). We say that sf{x) G M" is a subgradient of the function F at x E dom(F) 
when the following relation holds: 

F{x) + — x) < F{x) for all x G dom(F). (1) 

The set of all subgradients of F at x is denoted by dF{x) (see [2]). 

In our development, the properties of the projection operation on a closed convex set 
play an important role. We write dist{x,X) to denote the standard Euclidean distance 
of a vector x from a set X, i.e., 

distfx, X) = inf llx — xll. 

Let X be a nonempty closed convex set in M". We use Px[x] to denote the projection 
of a vector x on set X, i.e., 

Px[3S] = argmin ||x — x||. 

We will use the standard non-expansiveness property of projection, i.e., 

— < — y\\ for any x and y. (2) 

We will also use the following relation between the projection error vector and the feasible 
directions of the convex set X: for any x G M", 

\\Px[x] - yf < \\x - yf - \\Px[x] - xf for all ?/ G X. (3) 
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2 The Model 



2.1 Optimization Model 

We consider a network that consists of a set of nodes (or agents) Ai = {l,...,m}. 
We assume that each agent i is endowed with a local objective (cost) function /j and 
a local constraint function Xj and this information is distributed among the agents, 
i.e., each agent knows only his own cost and constraint component. Our objective is to 
develop distributed algorithms that can be used by these agents to cooperatively solve 
the following constrained optimization problem: 

minimize Yl'^LiM^) (4) 
subject to X G fl™ ^Xj, 

where each /j : — )■ M is a convex (not necessarily differentiable) function, and each 
Xi C M" is a closed convex set. We denote the intersection set hy X = fl^j^Xj and 
assume that it is nonempty throughout the paper. Let / denote the global objective, 
that is, f{x) = Yl^i fi{^)i /* denote the optimal value of problem Q, which we 
assume to be finite. We also use X* = {x E X : f{x) = /*} to denote the set of optimal 
solutions and assume throughout that it is nonempty. 

We study a distributed multi-agent subgradient method, in which each agent i main- 
tains an estimate of the optimal solution of problem ^ (which we also refer to as the 
state of agent i), and updates it based on his local information and information ex- 
change with other neighboring agents. Every agent i starts with some initial estimate 
Xi{0) G Xi. At each time k, agent i updates its estimate according to the following: 



Xi{k + 1) = Px, 



(5) 



''^^aij{k)xj{k) — a{k)di{k) 
.j=i 

where Px^ denotes the projection on agent i constraint set X^, the vector [aij(fc)]jg>j 
is a vector of weights for agent i, the scalar a{k) > is the stepsize at time k, and 
the vector di{k) is a subgradient of agent i objective function fi{x) at his estimate 
Vi{k) = J2Y=ia'ij{k)xj{k). Hence, in order to generate a new estimate, each agent 
combines the most recent information received from other agents with a step along 
the subgradient of its own objective function, and projects the resulting vector on its 
constraint set to maintain feasibility. We refer to this algorithm as the projected multi- 
agent subgradient algorithm^ Note that when the objective functions fi are identically 
zero and the constraint sets Xi = for alH G A^, then the update rule ^ reduces to 
the classical averaging algorithm for consensus or agreement problems, as studied in [7] 
and [H]. 

In the analysis of this algorithm, it is convenient to separate the effects of different 
operations used in generating the new estimate in the update rule ([s]). In particular, we 



^See also |19j where this algorithm is studied under deterministic assumptions on the information 
exchange model and the special case Xi ~ X for all i. 
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rewrite the relation in Eq. (|5]) equivalently as follows: 

m 

Mk) = ^aij{k)xj{k), (6) 

Xi{k + 1) = Vi{k)-a{k)di{k) + ei{k), (7) 
ei{k) = PxMk) - a{k)d,{k)] - (v,{k) - a{k)di{k)y (8) 

This decomposition allows us to generate the new estimate using a linear update rule in 
terms of the other agents' estimates, the subgradient step, and the projection error Cj. 
Hence, the nonlinear effects of the projection operation is represented by the projection 
error vector Cj, which can be viewed as a perturbation of the subgradient step of the 
algorithm. In the sequel, we will show that under some assumptions on the agent weight 
vectors and the subgradients, we can provide upper bounds on the projection errors as 
a function of the stepsize sequence, which enables us to study the update rule ([s]) as an 
approximate subgradient method. 

We adopt the following standard assumption on the subgradients of the local objec- 
tive functions 

Assumption 1. (Bounded Subgradients) The subgradients of each of the fi are uniformly 
bounded, i.e., there exists a scalar L > such that for every i ^ A4 and any x G M", we 
have 

\\d\\<L for alld e dfi{x). 



2.2 Network Communication Model 

We define the communication matrix for the network at time k as A{k) = [cLij{k)]ij^M- 
We assume a probabilistic communication model, in which the sequence of communica- 
tion matrices A{k) is assumed to be Markovian on the state variable x{k) = [xi{k)]ifzM G 
M"^™. Formally, let be an independent sequence of random variables defined 

in a probability space P) = Y[T=o(^' ^ ^ where {{Q' , J^' , P')k}keN consti- 

tutes a sequence of identical probability spaces. We assume there exists a function 
^ : M"^'" xn' ^ M™^"* such that 

A{k) = tlj{x{k),n{k)). 

This Markovian communication model enables us to capture settings where the agents' 
ability to communicate with each other depends on their current estimates. 

We assume there exists some underlying communication graph {Ai,S) that repre- 
sents a 'backbone' of the network. That is, for each edge e & S, the two agents linked 
by e systematically attempt to communicate with each other [see Eq. ^ for the pre- 
cise statement]. We do not make assumptions on the communication (or lack thereof) 
between agents that are not adjacent in {A4,S). We make the following connectivity 
assumption on the graph {M,£). 
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Assumption 2 (Connectivity). The graph {Ai,S) is strongly connected. 

The central feature of the model introduced in this paper is that the probabihty of 
communication between two agents is potentially small if their estimates are far apart. 
We formalize this notion as follows: for all G all A; > and all x G M*"^", 

Pittijik) > 7|x(/c) = x)> min Id, .._ ^_ ..A , (9) 

where K and C are real positive constants, and 6 G (0, 1]. We included the parameter 
6 in the model to upper bound the probability of communication when \\xi — xjW" is 
small. This model states that, for any two nodes i and j with an edge between them, 
if estimates Xi{k) and Xj{k) are close to each other, then there is a probability at least 
S that they communicate at time k. However, if the two agents are far apart, the 
probability they communicate can only be bounded by the inverse of a polynomial of 
the distance between their estimates — If the estimates were to represent 

physical locations of wireless sensors, then this bound would capture fading effects in 
the communication channel. 

We make two more technical assumptions to guarantee, respectively, that the com- 
munication between the agents preserves the average of the estimates, and the agents 
do not discard their own information. 

Assumption 3 (Doubly Stochastic Weights). The communication matrix A{k) is doubly 
stochastic for all k > 0, i.e., for all k >0, a.ij{k) > for alli,j G Ai, and Xll^i = 
1 for all j (z M. and X^jLi ^iji^) — 1 /^'^ all i E Ai with probability one. 

Assumption 4 (Self Confidence). There exists 7 > such that an > 7 for all agents 
i E Ai with probability one. 

The doubly stochasticity assumption on the matrices A{k) is satisfied when agents 
coordinate their weights when exchanging information, so that aij{k) = aji{k) for all 
i,j E Ai and k > Oj^ The self-confidence assumption states that each agent gives a 
significant weight to its own estimate. 



2.3 A Counterexample 

In this subsection, we construct an example to demonstrate that the algorithm defined 
in Eqs. (|6])-([8]) does not necessarily solve the optimization problem given in Eq. 
The following proposition shows that there exist problem instances where Assumptions 
[T]|4]hold and Xi = X for alH G A^, however the sequence of estimates Xi{k) (and the 
sequence of function values f{xi{k))) diverge for some agent i with probability one. 

Proposition 1. Let Assumptions^ \^ anc?!^ hold and let Xi = X for all i E Ai. Let 

{xi{k)} be the sequences generated by the algorithm ([^-(^. Let C > 1 in Eq. and let 

•^This will be achieved when agents exchange information about their estimates and "planned" 
weights simultaneously and set their actual weights as the minimum of the planned weights; see [T5] 
where such a coordination scheme is described in detail. 
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the stepsize be a constant value a. Then, there does not exist a bound M{m, L,a) < oo 
such that 

liminf \fix,{k)) - f*\ < M{m, L, a) 
with probability 1, for all agents i E M.. 

Proof. Consider a network consisting of two agents solving a one-dimensional mini- 
mization problem. The first agent's objective function is = —a;, while the sec- 
ond agent's objective function is /2(a;) = 2x. Both agents' feasible sets are equal to 
Xi = X2 = [0, 00). Let xi(0) > X2(0) > 0. The elements of the communication matrix 
are given by 

7, with probability min \,^^k)X{k)\o } 
0, with probability 1 - min \x,(k)-x,{k)\o } ^ 

for some 7 G (0, 1/2] and 6 G [1/2, 1). 

The optimal solution set of this multi-agent optimization problem is the singleton 
X* = {0} and the optimal solution is /* = 0. We now prove that limfc_>.oo = 00 

with probability 1 implying that limfc^oo Ifi^iik)) — f*\ = 00. 

From the iteration in Eq. we have that for any k, 

Xi{k + 1) = ai^i{k)xi{k) + ai^2{k)x2{k) + a (10) 
X2{k + 1) = max{0, a2,i(/i;)a;i(A;) + a2,2(^)2;2(^) — 2a}. (11) 

We do not need to project xi{k + l) onto Xi = [0, 00) because Xi{k + 1) is non-negative if 
Xi{k) and X2{k) are both non- negative. Note that since 7 < 1/2, this iteration preserves 
xi{k) > X2{k) > for all keN. 

We now show that for any A; G N and any Xi{k) > X2{k) > 0, there is probability at 
least e > that the two agents will never communicate again, i.e., 

P{ai^2{k') = a2,i{k') = for all k' > k\x{k)) > e > 0. (12) 

If the agents do not communicate on periods k,k + 1, k + j ~ 1 for some j > 1, then 

xi{k + j) - X2{k + j) = {xi{k + j) - xi{k)) + (xi(fc) - X2{k)) + {x2{k) - X2{k + j)) 

> aj + + 0, 



from Eqs. (10) and (11) and the fact that Xi{k) > X2{k). Therefore, the communication 
probability at period k + j can be bounded by 

P{ai,2{k + j) = 0\x{k), ai,2(A;') = for all k' e {k, k + j - 1}) > 1 - mm{5, (aj)-^}. 
Applying this bound recursively for all j > k, we obtain 

P(ai_2(A;') = for all k' > k\x{k)) 

00 

= Yl P(ai,2(fc + j) = 0\x{k), ai,2(fc') = for all k' G {k, ...,k + j - 1}) 

j=0 

00 

> J](l-min{5, (ajr""}) 

j=0 
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for all k and all xi{k) > X2{k). We now show that YlJLo — niin{(5, (aj) > if 



C > 1. Define the constant K 



1 -\ 

2C7 



Since 5 > 1/2, we have that (aj) < 5 for 



j > K. Hence, we can separate the infinite product into two components: 



J] (1 - min{(5, (ajr^}) > 

3=0 



n (l-min{5, (aj)-^}) 



n (1 - ("^)"°) 



Note that the term in the first brackets in the equation above is a product of a finite 
number of strictly positive numbers and, therefore, is a strictly positive number. We, 
thus, have to show only that nj>j? ~ ('^j) *") > 0- We can bound this product by 



n (1 - ("jr"^) = exp log n (1 - ("•^■)"'') 

3>K \ \j>K 

( 



exp 



log (1 - («jr'') > exp ^ -(ajr''log(4) 



where the inequality follows from log(a;) > (x — 1) log(4) for all x G [1/2, 1]. Since C > 1, 
the sum X]j>7?('^i)~'" 1^ finite and Hjlo (-'- ~ niin{5, (aj)^'"}) > 0, yielding Eq. ( |T2| . 

Let ii"* be the (random) set of periods when agents communicate, i.e., ai,2(^) = 
02,i(^) = 7 if and only if A; G K* . For any value k G if* and any x\(}z) > X2{k), there is 
probability at least e that the agents do not communicate after k. Conditionally on the 
state, this is an event independent of the history of the algorithm by the Markov property. 
If K* has infinitely many elements, then by the Borel-Cantelli Lemma we obtain that, 
with probability 1, for infinitely many fc's in K* there is no more communication between 
the agents after period k. This contradicts the infinite cardinality of K*. Hence, the two 
agents only communicate finitely many times and \imk^ooXi{k) = oo with probability 
1. ■ 

The proposition above shows the algorithm given by Eqs. ([gJ-Q does not, in general, 
solve the global optimization problem Q. However, there are two important caveats 
when considering the implications of this negative result. The first one is that the 
proposition only applies if C > 1. We leave it is an open question whether the same 
proposition would hold if C < 1. The second and more important caveat is that we 
considered only a constant stepsize in Proposition [1} The stepsize is typically a design 
choice and, thus, could be chosen to be diminishing in k rather than a constant. In 
the subsequent sections, we prove that the algorithm given by Eqs. ([6])-([8]) does indeed 
solve the optimization problem of Eq. Q, under appropriate assumptions on the stepsize 
sequence. 
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3 Analysis of Information Exchange 



3.1 The Disagreement Metric 

In this section, we consider how some information that a given agent i obtains at time 
s affects a different agent j's estimate Xj{k) at a later time k > s. In particular, 
we introduce a disagreement metric p{k, s) that establishes how far some information 
obtained by a given agent at time s is from being completely disseminated in the network 
at time k. The two propositions at the end of this section provide bounds on p{k, s) 
under two different set of assumptions. 

In view of the the linear representation in Eqs. ([6])-([8]), we can express the evolution 
of the estimates using products of matrices: for any s > and any k > s, we define the 
transition matrices as 

s) = A{s)A{s + 1) ■ ■ ■ A{k - l)A{k) for all s and k with k > s. 

Using the transition matrices, we can relate the estimates at time k to the estimates at 
time s < k as follows: for all i, and all k and s with k > s, 

m km 

x.{k + l) = J2[ms)hx,{s) - J2l'^{k,r)].Mr-l)dj{r-l)-a{k)d.{k) 

j=l r=s+l j=l 

k m 

+ 5^ $^[$(A:,r)].,e,(r-l) + e.(fc). (13) 

r=s+l j=l 

Observe from the iteration above that [^{k,s)]ij determines how the information 
agent i obtains at period s — 1 impacts agent j's estimate at period k + 1. If [^{k, s)]ij = 
1/m for all agents j, then the information agent i obtained at period s — 1 is evenly 
distributed in the network at time k + 1. We, therefore, introduce the disagreement 
metric p, 

[<l>{k,s)],j - - 



p{k, s) = max 



m 



for all A; > s > 0, (14) 



which, when close to zero, establishes that all information obtained at time s — 1 by all 
agents is close to being evenly distributed in the network by time A; + 1. 



3.2 Propagation of Information 

The analysis in the rest of this section is intended to produce upper bounds on the 
disagreement metric p{k,s). We start our analysis by establishing an upper bound on 
the maximum distance between estimates of any two agents at any time k. In view 
of our communication model [cf. Eq. ([o])], this bound will be essential in constructing 
positive probability events that ensure information gets propagated across the agents in 
the network. 
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Lemma 1. Let Assumptions^ and^hold. Let Xi{k) be generated by the update rule 

Then, we have the following upper bound on the norm of the difference between the 
agent estimates: for all k > 0, 



fc— 1 fc— 1 m 

max \\xi{k) - Xhik)\\ < A + 2mL^a{r) + 2^^||ej 

r=0 r=0 j=l 



where A = 2mma.Xj^M '^^'^ ^j(^) denotes the project 



ton error. 



Proof. Letting s = in Eq. (13) yields, 



.{k) = j2mk-i,o)].,xM 



fc— 1 m 

r=l j=l 

fc— 1 m 

+ J2 Yl'^ik - 1, r)],,e,(r - 1) + e,{k - 1). 

r=l j=l 

Since the matrices A{k) are doubly stochastic with probability one for all k (cf. As- 
sumption [3]), it follows that the transition matrices $(/c, s) are doubly stochastic for all 
k > s > 0, implying that every entry [$(/c, s)]ij belongs to [0, 1] with probability one. 
Thus, for all k we have, 

m fc— 1 m 

lk.(fc)||<^||a;,(0)|| + ^^a(r-l)||rf,(r-l)||+a(fc-l)||rf,(fc-l)|| 

j=l r=l j=l 

fc— 1 m 

+ EE 1)11 + 11^^(^-1)11- 

r=l j=l 

Using the bound L on the subgradients, this implies 

m fc— 1 fc— 1 m 

\\xm\ < E 11^^- wii + E^^"W + E E ii^^-wii- 

j=l r=0 r=0 j=l 

Finally, the fact that \\xi{k) —Xh{k)\\ < \\xi{k)\\ + \\xh{k)\\ for every i, /i G A^, establishes 
the desired result. ■ 

The lemma above establishes a bound on the distance between the agents' estimates 
that depends on the projection errors Cj, which are endogenously determined by the 
algorithm. However, if there exists some M > such that ||ej(A;)|| < Ma{k) for alH G 
and all k > 0, then lemma above implies that, with probability 1, maxi^h&M W^iik) — 
Xh{k)\\ < A + 2m{L + M) Yl!l=o (^{^)- Under the assumption that such an M exists, we 
define the following set for each A; G N, 



= <! s G M"^" I max -x?,(A;)|| < A + 2m(L + M)y a(r) I . (15) 

' %,h£M ^ — ^ I 



r=0 
11 



This set represents the set of agent states which can be reached when the agents use the 
projected subgradient algorithm. 

We next construct a sequence of events, denoted by G{-), whose individual occur- 
rence implies that information has been propagated from one agent to all other agents, 
therefore, implying a contraction of the disagreement metric p. 

We say a link (j, z) is activated at time k when aij{k) > 7, and we denote by £{k) 
the set of such edges, i.e., 

S{k) = {{j,t) |a,,(A;)>7}. 

Here we construct an event in which the edges of the graphs £{k) are activated sequen- 
tially over time k, so that information propagates from every agent to every other agent 
in the network. 

To define this event, we fix a node w & Ai and consider two directed spanning trees 
rooted at w in the graph {Ji4,£): an in-tree Tj„_^ and an out-tree Tout,w In ^m,w) there 
exists a directed path from every node i w to w; while in Tout,w, there exists a directed 
path from w to every node i ^ w. The strongly connectivity assumption imposed on 
{Ai,£) guarantees that these spanning trees exist and each contains m — 1 edges (see 

i)- 

We order the edges of these spanning trees in a way such that on any directed path 
from a node i w to node w, edges are labeled in nondecreasing order. Let us represent 
the edges of the two spanning trees with the order described above as 

62, • • • , Tout,w — {fl, f2, ■ ■ ■ , fn-l}- (16) 

For the in-tree Tj„ „,, we pick an arbitrary leaf node and label the adjacent edge as Ci, 
then we pick another leaf node and label the adjacent edge as 62; we repeat this until 
all leaves are picked. We then delete the leaf nodes and the adjacent edges from the 
spanning tree Tin^r, and repeat the same process for the new tree. For the out-tree 
Tout,w, we proceed as follows: pick a directed path from node w to an arbitrary leaf 
and sequentially label the edges on that path from the root node w to the leaf; we then 
consider a directed path from node w to another leaf and label the unlabeled edges 
sequentially in the same fashion; we continue until all directed paths to all the leaves 
are exhausted. 

For all / = 1, . . . , m — 1, and any time k > 0, consider the events 

Bi{k) = {tuen\a,^{k + l-l)>j}, (17) 
Di{k) = {iu en\ af^{k + {m-l) + l-l)> 7}, (18) 



and define, 



m—l 



G{k)= fl (^Bi{k) n Di{k)y (19) 
1=1 

For all / = 1, ... ,m — 1, Bi{k) denotes the event that edge ci G T^^^ is activated at 
time k + I — 1, while Di{k) denotes the event that edge fi G Tout,w is activated at time 
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k + {m — 1) + 1 — 1. Hence, G{k) denotes the event in which each edge in the spanning 
trees Tj„ „, and Tout,w are activated sequentially following time fc, in the order given in 



Eq. (16). 



The following result establishes a bound on the probability of occurrence of such a 
(j(-) event. It states that the probability of an event G{-) can be bounded as if the link 
activations were independent and each link activation had probability of occurring at 
least 

mm < 0, 



(A + 2m(L + M) YHIT"'^ <r))c 

where the k + 2m — 3 follows from the fact that event G{-) is an intersection of 2(m — 1) 
events occurring consecutively starting at period k. 

Lemma 2. Let Assumptions [IJ and hold. Let A denote the constant defined in 
Lemma^ Moreover, assume that there exists M > such that ||ej(fc)|| < Ma{k) for all 
i and k > 0. Then, 

(a) For all s eN, k > s, and any state x G Rm{s), 

N 2(m-l) 

K w / 

P{G{k)\x{s) =x)> min<' 5, 



(A + 2m(L + M) YUtt"^ 
(h) For all k > 0, u > 1, and any state x G Ruik), 



fu-l 



P U G{k + 2{m-l)l) 



x{k) = X 



K 

> 1 — I 1 — min I 5 



2(m-l)\ " 



(A + 2m(L + M) (^{r))c 



Proof, (a) The proof is based on the fact that the communication matrices A{k) are 
Markovian on the state for all time > 0. First, note that 



^m— 1 



P{G{k)\x{s)=x) = p{ fl {Bi{k)^Di{k) 

\ 1=1 

/m-l 



1=1 



x[s] = X 



'm— 1 



XiS] = X 



1=1 



m— 1 



X[S] = X 



1=1 



(20) 



To simplify notation, let W = 2m{L + M). We show that for all k > s, 



'm—l 



inf pin Bi{k) 



x(s) = X > min < 6, 



K 



(m-l) 



(21) 
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We skip the proof of the equivalent bound for the second term in Eq. (20) to avoid 
repetition. By conditioning on x{k) we obtain for all k > s, 



inf P n Bi{k) 

x&Rm{s) * ' ' 



1=1 



XiS) = X 



1 



inf / ^ ( n ^'(^) 



x{k) = x',x{s) = X ) dP{x{k) = x'\x{s) = x). 



Using the Markov Property, we see that conditional on x{s) can be removed from the 
right-hand side probability above, since x{k) already contains all relevant information 
with respect to {yi^r^^Biik). By the definition of Rm{-) [see Eq. (15)], if x{s) G Rm{s), 
then x{k) G RM{k) for all k > s with probability 1. Therefore, 



^m— 1 



inf pin Bi{k) 



1=1 



'm—l 



XiS) = X 



> inf pin Bi{k) 



, 1=1 



x{k) = x'] . (22) 



By the definition of Bi{k), 



'm—l 



inf pin Bi{k) 



1=1 



x{k) = X 



(23) 



' m—l 



inf P{aeM > l\x{s) =x)Pi n Bi{k) 
xei?M(fc) 'III 



Define 



Q{k) = min < 5, 



. 1=2 



K 



a-eiik) > 7, x{k) = X 



C ( ' 



and note that, in view of the assumption imposed on the norm of the projection errors 
and based on Lemma [T| we get 



k-l 



max \\xi{k) - Xh{k)\\ < A + a{r). 

>.,heM 



r=0 



Hence, from Eq. ^ we have 



P{aij{k) > j\x{k) =x)> Q{k). 



(24) 



Thus, combining Eqs. (23) and (24) we obtain, 

^ m—l 



inf pin Bi{k) 

x&RM{k) > ' ' 



1=1 



' m—l 



x(k) = x\ > Qik) inf pin BAk) 

I ' x€RM(k) » I ' 



. 1=2 



o.ei{k) > 'j,x{k) = xj . 

(25) 
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By conditioning on the state x{k + 1), and repeating the use of the Markov property and 
the definition of Ruik + 1), we can bound the right-hand side of the equation above, 



(m-l 
1=2 



' m— 1 



inf / P ( n Bi{k) 

x&RM{k) J^, y j^^ 

(m—l 
n 



x{k + 1) = x'^ dP{x{k + 1) = x\aei{k) > 7, x{k) = x) 
x{k + l)=x' ] . (26) 



Combining Eqs. (23), (25) and (26), we obtain 



(m— 1 
1=1 



'm—l 



x(k)=x\>Q(k) inf P i H Bdk) 
J 5rei?A/(fc+i) y\J^ 



x{k + 1) = x' \ . 



Repeating this process for all / = 1, ...,m — 1, this yields 



(m—l 
1=1 



m—l 



x{k) =x\>WQ{k + l-l). 



Since Q is a decreasing function, H/^i Qi^ + / — 1) > Q{k + 2m — 3)™ ^. Combining 



with Eq. (22), we have that for all > s 

m—l 



inf P n Bi{k) 



x{s) =x] > Q{k + 2m - 3) 



m—l 



producing the desired Eq. (21). 



(b) Let G'^{k) represent the complement of G{k). Note that 



''u-l 



P [j G{k + 2{m - 1)1) 



'u-l 



xik) = ^ = 1 - P Pi G\k + 2(m - \)l) 



x{k) = X 



By conditioning on G'^{k), we obtain 



^u-l 



P f]G'{k + 2{m-l)l) 



x{k) = X 



'u-l 



P {G\k)\x{}z) = x) P Pi G^ik + 2(m - 1)/) 



G\k),x{k) = x 



We bound the term P {G^{k)\x{k) = x) using the result from part (a). We bound the 
second term in the right-hand side of the equation above using the Markov property and 
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the definition of R]\i{-), wliicli is tlie same technique from part (a) 



sup P Pi G"^(A; + 2(m - 1)0 



J=l 



G^{k),x{k) =x 



sup [ P (Pi G\k + 2(m - 1)/) 

x(iRM{k)Jx' \i=i 



x{k + 2(m - 1)) = x' X 



dP{x{k + 2(m - 1)) = x'\G\k),x{k) = x) 



< sup P n + 2(m - 1)/) 

xe/?M(fc+2(m-l)) 



a;(/c + 2(m — 1)) = x' 



The result follows by repeating the bound above u times. ■ 

The previous lemma bounded the probability of an event G{-) occurring. The fol- 
lowing lemma shows the implication of the event G{-) for the disagreement metric. 

Lemma 3. Let Assumptions\^\^and\^hold. Lett be a positive integer, and let there be 
scalars s < Si < S2 < ■ ■ ■ < St < k, such that s^+i — Sj > 2(m — 1) for alii = 1, . . . , t — 1 . 
For a fixed realization u E Q, suppose that events G{si) occur for each i = 1, . . . ,t. 
Then, 

P(k,s)<2(l + :^) (1-0*'-))*. 
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We skip the proof of this lemma since it would mirror the proof of Lemma 6 in [15] . 
3.3 Contraction Bounds 

In this subsection, we obtain two propositions that establish contraction bounds on the 
disagreement metric based on two different sets of assumptions. For our first contraction 
bound, we need the following assumption on the sequence of stepsizes. 

Assumption 5. (Limiting Stepsizes) The sequence of stepsizes {<y{k)}kem satisfies 

lim log^(A;)a(A;) = for all p < 1. 

k—>oo 

The following lemma highlights two properties of stepsizes that satisfy Assumption 
|5j they are always square summable and they are not necessarily summable. The con- 
vergence results in Section |4] require stepsizes that are, at the same time, not summable 
and square summable. 

Lemma 4. Let {a{k)}k£fq be a stepsize sequence that satisfies Assumption^ Then, 
the stepsizes are square summable, i.e, YlT=o'^'^i^) ^ Moreover, there exists a 
sequence of stepsizes {a{k)}kefi that satisfies Assumption^ and is not summable, i.e., 

Er=o«(^) = 



16 



Proof. From Assumption |5| with j9 = 0, we obtain that there exists some K E N such 
that a{k) < 1/k for all k > K. Therefore, 

oo K—1 oo _ 9 

a^(k) a^(k) + — <K max a^(fc) H < oo. 

^fc^ fee{o,...,i^-i} 6 

A;=0 k=0 k=K i > ' j 

Hence, {a(fc)}fc6N is square summable. Now, let a{k) = (fc_|_2) iog(fc+2) k E N. This 

sequence of stepsizes satisfies Assumption [s] and is not summable since for all K' E N 

K' 



J]«(A;) >log(log(K' + 2)) 



k=0 

and limi^/^oo log(log(/r' + 2)) = oo. ■ 

The following proposition is one of the central results in our paper. It establishes, 
first, that for any fixed s, the expected disagreement metric E[p{k, s)\x{s) = x] decays 
at a rate of e^*^"'^ as k goes to infinity. Importantly, it also establishes that, as s grows, 
the contraction bound for a fixed distance k — s decays slowly in s. This slow decay is 
quantified by a function /3(s) that grows to infinity slower than the polynomial s'^ for 
any g > 0. 

Proposition 2. Let Assumptions^ \^\^^ and\^hold. Assume also that there exists 
some M > such that ||ej(A;)|| < Ma{k) for all i E Ai and k Efi. Then, there exists a 
scalar fi > 0, an increasing function f3{s) : N — t- IR+ and a function S{q) : N — )■ N such 
that 

I3{s) < for allq>0 and all s > S{q) (27) 

and E[p{k, s)\x{s) = x]< l3{s)e-''^^' for allk>s>0, xE Rm{s). (28) 



Proof. Part 1. The first step of the proof is to define two functions, g{k) and w{k), that 
respectively bound the sum of the stepsizes up to time k and the inverse of the probability 
of communication at time k, and prove some limit properties of the functions g{k) and 
w{k) [see Eqs. (29) and (31)]. Define g{k) : IR+ — )■ IR+ to be the linear interpolation of 

Ei=o«(^)' i-e, 

Lfcj 

g(k) = 5]]tt('^) + {k~ [k\)a{k - [k\ + 1). 

r=0 

Note that g is differentiable everywhere except at integer points and g'{k) = a{k— \_k\ + 
1) = «([/;;]) at k ^N. We thus obtain from Assumption |5] that for all p < 1, 

(29) 



lim k\ogP{k)g'{k) = lim \k] \ogP{\k])a{\k]) = 0. 

fc^oo.fc^N fc— s>oo 



Define w{k) according to 



w{k) 



(A + 2m(L + M)^(A;))2(™-i)^ 

ir2(m-l) • 



(30) 
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where A = 2mmaXjg_A4 ||3;j(0)|| and K and C are parameters of the communication 
model [see Eq. Q]. We now show that for any p < 1, 



lim k\ogP{k)w\k) = Q. 



(31) 



If limfc_^oo '2^(^) < oo, then the equation above holds immediately from Eq. (29). There- 
fore, assume limfc^oo w{k) = oo. By L'Hospital's Rule, for any q > 0, 



lim 



w{k) 



lim 



kw'{k) 



k^oo,k<^n\og'^{k) g fc^oo,fc^N log^ ^{k)' 



(32) 



At the same time, if we take w{k) to the power 2{m~-i)c before using L'Hospital's Rule, 
we obtain that for any g > 0, 



w{k) 

k->'cZ'',k0i \\og'^{k) 



lim 



2(m-l)C 



lim 



A + 2m(L + M)^(A;) 



K^l'^ k'^oo,k0i lQg2(m-i)C 

Am{m - 1){L + M)C ^.^ kg\k) _^ 



where the last equality follows from Eq. (29). From the equation above, we obtain 

-1 2(m-l)C 



w{k) 

fc^oo.fe^N \og'^{k) 



lim 



w{k) 

A;-^oo,fc^N \^log''(A;) 



lim 



2(m-l)C 



0, 



(33) 



which combined with Eq. (32), yields the desired Eq. (31) for any p = 1 — g < 1. 



Part 2. The second step of the proof involves defining a family of events {Hi{s)}i^s 
that occur with probability at least > 0. We will later prove that an occurrence of 
Hi{s) implies a contraction of the distance between the estimates. Let hi{s) = i+ \w{2s)'] 
for any i,s E N, where w{-) is defined in Eq. (30). We say the event Hi{s) occurs if 



one out of a sequence of G-events [see definition in Eq. (19)] starting after s occurs. In 
particular, Hi{s) is the union of hi{s) G-events and is defined as follows. 



HAs] 



U G + 2(m - 1) (^J - 1 + 5^ K{s)j j for 



all 2, s G N, 



where Ylr=ii') ~ 0- Figure 1 for a graphic representation of the Hi{s) events. We 
now show P{Hi{s)\x{s) = x) for all i,s eN and all x G Rm{s) [see definition of Rm{s) in 
Eq. (15)]. From Lemma|2](a) and the definition of w(-), we obtain that for all a; G -R 

1 



P{G{s)\x{s) =x)> min j^^C--!) 



'm{s), 



w 



[s + 2m- 3) 
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_Jl_ 



_i_ 



G[a + hi{a)2(m - 1)) 









• • • 


N 






• • • 






(m - 1) 


s + (fti(s) 


+ l)2Cm - 1 


) 


> 



s + ki{s)2{m - 1) 



s + {hi{3) + h2{s))2{m - 1) 




B,„ 1(5) 



T V V 



s+1 s + 2 



5 + 2(m - 1) 



Figure 1: The figure illustrates the three levels of probabilistic events considered in the 
proof: the events Bi{s) and -Dz(s), which represent the occurrence of communication 
over a link (edge of the in-tree and out-tree, respectively); the events G{s) as defined 



in (19), with length 2(m — 1) and whose occurrence dictates the spread of information 
from any agent to every other agent in the network; the events Hi{s) constructed as the 
union of an increasing number of events G{s) so that their probability of occurrence is 
guaranteed to be uniformly bounded away from zero. The occurrence of an event Hi{s) 
also implies the spread of information from one agent to the entire network and, as a 
result, leads to a contraction of the distance between the agents' estimates. 
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Then, for all s,i eN and all x G -RAf(s), 

U G s + 2(m-l) + J]/,, 

i=l \ \ r=l 

> 1-1 l-min<' 



hi{s) 



w[s\ 2(m - 1) XlUi ^''(■5' 

where the inequality follows from Lemma [2]^b) and the fact that w(-) is a non-decreasing 
function. Note that > r for all r and s, so that s + 2(m — 1) X]r=i ^r(s) > ■ Let 

/ be the smallest i such that w(i^) > ^~'^'<^~^^ . We then have that for all i > I, all s 
and all x G Rm(s), 



P{Hi{s)\x{s) =x) > 1 



/ 1 

-i 

y w f s + 2(m - 1) ^r(s) 



Let / be the maximum between / and the smallest i such that w{i'^) > 1. Using the 
inequality (1 — 1/x)^ < for all x > 1, and multiplying and dividing the exponent in 

the equation above hy w (^s + 2(m — 1) J2l=i ^r(s) j we obtain 



hi(s) 



PiHi{s)\xis) = x) > 1 - e »(=+2('"-i)S^=i'^'-(=)) 

for all i > I, all s and all x G Rm{s). By bounding hr{s) < hi{s) and replacing 

hi{s) = i + \w{2s)], we obtain 

_ i+fii){2s)l _ i+iii(2s) 

P(-ffj(s) |x(s) = X) > 1 — e »''(s + 2(m-l)(i^+iriJ'(2s)l)) > ]_ _ g ii,(s + 2(m--l){i2+itu(2s) + 0) 

We now show there exists some / such that 

i + w(2s) _ 

1 - e »(«+2(m-i)(i2+,„(2s)+i)) jg increasing in i for alH > 7, s G M. (34) 
The function above is increasing in i if , , — ^tw'?!''- to ^ , ■^^ is increasing in i. The 

w(s+2{m—l){i''+iw(2s)+i)) 

partial derivative of this function with respect to i is positive if 
w{s + 2(m - + iw{2s) + i)) - 

2{m - l){2f + i + 3iw{2s) + w{2s) + w\2s))w' [s + 2{m - + iw{2s) + i)) > 

at all points where the derivative w'{-) exists, that is, at non-integer values. If 2 > J, 
then w [s + 2{m — + iw{2s) + i)) > 1 for all s and it is thus sufficient to show 

2(m - l){2i'^ + i + 3iw{2s) + w{2s) + w^{2s))w' {s + 2{m - + iw{2s) + i)) < I 
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in order to prove that Eq. (|34j) hold. The equation above holds if 

2(m - l){3i'^ + 4iw{2s) + w^{2s))w' (2(m - + iw{2s)) + s) < 1. 



From Eq. (31) with p = 1/2, we have that there exists some N such that for all x > N, 



w'(x) < 



. For i > N and any s G N, 



2(m - l)(3i2 + Aiw{2s) + w'^{2s))w' (2(m - + iw{2s)) + s) 

2{m - l)(3i2 + Aiw{2s) + w'^{2s)) 



< 



< 



(2(m - l)(4i2 + 4iw(2s)) + 4s)^log(2(m - + iw{2s) + s) 
4i2 + 4iw(2s) + ^Sv/log(i2)' 



The term above is less than or equal to 1 if we select i large enough such that s ^yiogiW} > 
'Up (2s) for all s G N [see Eq. (33) with q < 1/2], thus proving there exists some / such 



that Eq. (34) holds. Hence, we obtain that for all i, s G N and all x G Rm{s), 



P{H,{s)\x{s) = x) > 



1 




hj{s) ' 



mm_ < 1- I 1-min^ 

I I I m; ( s + 2(m - 1) J2i=i Kis 



Since P{Hi{s)\x{s) = x) > for all i,s E N and all x G Rm{s), to obtain the uniform 
lower bound on P{Hi{s)\x{s) = x) > > 0, it is sufficient to show that for all i G 
{1, /} and all x G Rm{s), 

lim P{Hi{s)\x{s) =x)>0. 

s—>ca 

Repeating the steps above, but constraining s to be large enough instead of i, we obtain 
there exists some S such that for all s > 5*, alH G N and x G Rm{s), 

i + w(2s) 

P{Hi{s)\x{s) =x)>l- e"'"('<+2(™-i)('"+-(2-)+o). 



Since there exists some S such that w{2s) < log(2s) for all s > S* [see Eq. (33) with 
g = 1], we obtain 



i + w(2s) 



P{Hi{s)\x{s) =x) >l-e »(-+2(— i)(«^+nog(2.)+o) _ 

for s > max{S,S} and all z G N and x G Rm{s). Note that for every i, there exists 
some S{i) such that for all s > S{i) the numerator is greater than the denominator in 
the exponent above. Therefore, for alH G N and x G Rm{s), 



lim P{Hi{s)\x{s) =x)>l- e-\ 
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Hence, there indeed exists some > such that P{Hi{s)\x{s) — x) > (f) for all i,s eN 
and X e Rm{s). 

Part 3. In the previous step, we defined an event Hi{s) and proved it had probability 
at least > of occurrence for any i and s. We now determine a lower bound on the 
number of possible i7-events in an interval {s, k}. The maximum number of possible 
H events in the interval {s, /c} is given by 

u{k, s) = max |i e N | s + 2(m ~ -*-) ^ ^»(*) - ^ 
Recall that hi{s) — [^(25)] <i-\- w{2s) + 1 to obtain 



u{k, s) > max i i e N | ^(i + ^(25) + 1) < 



i=l 



k — s 
2(m-l] 



By expanding the sum and adding (| + w(2s))^ to the left-hand side of the equation 
inside the maximization above, we obtain the following bound 

u{k, s) > max ^ i e N | + 3i + 2w{2s)t + ( ^ + w{2s)] < ^ " ^ 



3 k — s 

= max^ieNliH hw(2s)<\ '- 

' 2 V m — 1 



which yields the desired bound on u{k,s), 



<k,s)>^l^^-l-w{2s) 



(35) 



Part 4- We now complete the proof of the proposition. The following argument shows 
there is a high probability that several if-events occur in a given {s, k} interval and, 
therefore, we obtain the desired contraction. 

Let Ii{s) be the indicator variable of the event Hi{s), that is = 1 if Hi{s) occurs 
and /j(s) = otherwise. For any k > s > 0, any x G Rm{s) and any 6 > 0, the 
disagreement metric p satisfies 



E[p{k, s)\x{s) = x] = 

u{k,s) 

(s) = X, Ii{s) > 6u{k,s) 
1=1 



E 



E 



p{k,s) 
p(k,s) 



X[ 



u{k,s) 

x{s) — X, Ii{s) < du{k, s) 

1=1 



u(k,s) 

P ( Iiis)>Hk,s) 

i=l 
u{k,s) 

P \ h{s)<5u{k,s) 
1=1 



x{s) = x\ -\- 
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Since all the terms on the right-hand side of the equation above are less than or equal 
to 1, we obtain 



E[p{k,s)\x{s) = x]< 

u{k,s) 

[s) = X, > 6u{k, s) 



(36) 



E 



p{k,s] 



1=1 



'u{k,s) 

P I h{s)<5u{k,s) 



X[S] = X 



We now bound the two terms in the right-hand side of Eq. (36). Consider initially 
the first term. If /j(s) > 6u{k,s), then at least 6u{k,s) H-events occur, which by the 
definition of Hi{s) implies that at least 5u{k,s) G-events occur. From Lemma |3| we 
obtain 



E 



p{k,s] 



u{k,s) 

(s) = X, Ii(s) > 5u{k, s) 

i=l 



< 2 1 + 



2{m-l) 



7 



2(m-l)\'^"('=''5) 



(37) 



for all 5 > . We now consider the second term in the right-hand side of Eq. ( 36 ) . The 
events {Ii{s)}i=i^,,,^u(^k,s) all have probability at least > conditional on any x{s) G 
Rm{s), but they are not independent. However, given any x{s + hj{r)) G R{s + 
^11=1 ^il'^))' the event Ij{s) is independent from the set of events {/i(s)}j=i,...j_i by the 
Markov property. Therefore, we can define a sequence of independent indicator variables 
{Ji{s)}i=i,...,u{k,s) such P{Ji{s) = I) = (p and Ji{s) < Ii{s) for all i e {1, ...,u{k, s)} 
conditional on a;(s) G Rm(s). Hence, 



'u{k,s) 



1=1 



\ 

x{s) = X 









u{k,s) 



i=l 



(St 



for any 5 > and any x G -Rj\/(s). By selecting 6 = ^ and using Hoeffding's Inequality, 
we obtain 

u{k,s) 



u{k, s) 



E Ms) < I < 



e 2 



2^u{k,s) 



(39) 



i=l 



Plugging Eqs. (|37|, (|38]) and (|39]), with 6 = 0/2, into Eq. we obtain 

1 



E[p{k,s)\x{s) = x] < 2 1 + 



7 



2(m- 



_ 2(m-l)\ 



IT (1-7^""-") 



for all /c > s > and all x G Rm{s). This implies there exists some Pq,]Ii > such that 
p{k,s) < /iQC"'"!"*^'^''^^ and, combined with Eq. (35), we obtain there exist some K,p> 
such that 

E[p{k, s)\x{s) =x]< Ke^i""^^'^-^') for all A; > s > 0, x{s) G Rm{s). 

Let (3{s) = Ke^'^^'^^\ Note that (3{-) is an increasing function since w{-) is an increasing 
function. To complete the proof we need to show that /3 satisfies condition stipulated in 
Eq. (27). From Eq. (33), with g = 1, we obtain that 



lim 

s^oo,sif_n log(sj 



0. 
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Note that since w{-) is a continuous function, the hmit above also apphes over the 
integers, i.e., hms_5.oo j)^^ = 0. Since hms_s.oo(s), for any g > 0, we have 

= hm ^^^^^ = hm M^)+/i^(23) ^ log(^_ 

s^oo log(2s) s-5>oo g(— log(2) + log(2s)) s^oo log(s'^) 

Let S{q) be a scalar such that ^"pg'^fqj"' < 1 for all s > S{q). We thus obtain that j3{s) < s"^ 
for all s > S{q), completing the proof of the proposition. ■ 

The above proposition yields the desired contraction of the disagreement metric p, 
but it assumes there exists some M > such that ||ej(A;) || < Ma{k) for alH G and k G 
N. In settings where we do not have a guarantee that this assumption holds, we use the 
proposition below. Proposition [3] instead requires that the sets Xi be compact for each 
agent i. With compact feasible sets, the contraction bound on the disagreement metric 
follows not from the prior analysis in this paper, but from the analysis of information 
exchange as if the link activations were independent across time. 

Proposition 3. Let Assumptions \^ and ^ hold. Assume also that the sets Xi are 
compact for alii ^ Ai. Then, there exist scalars n, fi > such that for all x G IlieA^ -^i' 

E[p{k, s)\x{s) =x]< Ke-"'-''-'^ for allk>s>0. (40) 



Proof. From Assumption |2j we have that there exists a set of edges S of the strongly 
connected graph {M,S) such that for all {j,i) G all > and all x G M*"^", 

r K 

P{aij{k) > 7|x(fc) = x) > min < 5, 



I T' ■ T' • W ^ 



The function min |5, | is continuous and, therefore, it attains its optimum when 

minimized over the compact set Hjex -^iy i-^-, 

inf min 1 6, t. \ = min min 1 6, 



Since the function minj^, pr^-|p| is strictly positive for any x G M™^", we obtain 
that there exists some positive e such that 

r K 1 

e = inf min < S, -r— _ > > 0. 



Hence, for all {j,i) G all > and all x G HieA^ 

P{aij{k) > -i\x{k) =x)>e. (41) 

Since there is a uniform bound on the probability of communication for any given edge in 
8 that is independent of the state x{k), we can use an extended version of Lemma 7 from 



24 



|15j . In particular, Lemma 7 as stated in [T5] requires the communication probability 
along edges to be independent of x{k) which does not apply here, however, it can be 
extended with straightforward modifications to hold if the independence assumption 
were to be replaced by the condition specified in Eq. (41), implying the desired result. 



4 Analysis of the Distributed Subgradient Method 

In this section, we study the convergence behavior of the agent estimates {xi{k)} gen- 
erated by the projected multi-agent subgradient algorithm ([s]). We first focus on the 
case when the constraint sets of agents are the same, i.e., for all i, Xi = X for some 
closed convex nonempty set. In this case, we will prove almost sure consensus among 
agent estimates and almost sure convergence of agent estimates to an optimal solution 
when the stepsize sequence converges to sufficiently fast (as stated in Assumption [s]) . 
We then consider the case when the constraint sets of the agents Xi are different convex 
compact sets and present convergence results both in terms of almost sure consensus 
of agent estimates and almost sure convergence of the agent estimates to an optimal 
solution under weaker assumptions on the stepsize sequence. 

We first establish some key relations that hold under general stepsize rules that are 
used in the analysis of both cases. 

4.1 Preliminary Relations 

The first relation measures the "distance" of the agent estimates to the intersection set 
X = n™ ^Xj. It will be key in studying the convergence behavior of the projection errors 
and the agent estimates. The properties of projection on a closed convex set, subgra- 
dients, and doubly stochasticity of agent weights play an important role in establishing 
this relation. 

Lemma 5. Let Assumption^ hold. Let {xi{k)} and {ej(fc)} be the sequences generated 
by the algorithm (^-(^. For any z E X = r\^-^Xi, the following hold: 

(a) For all k > 0, we have 

m mm 

5^||x,(A; + l)-2f < J2\\xi{k)-z\\^ + a\k)Y,\\diikW 

i=l 1=1 i=l 

m m 

-2a{k) J2 {d^{k)'{vi{k) -z))~Y. W^^i^W- 

i=l i=l 

(b) Let also Assumption^hold. For all k>0, we have 

mm m 
i=l i=l i=l 

(42) 
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Moreover, for all k > 0, it also follows that 

mm m 

\\xj{k + 1) - < \\xj{k) - zf + a\k)mL^ + 2a{k)L ^ \\xj{k) - y{k)\\ 

-Mk) {f{y{k)) - f{z)) , ' ' (43) 

Proof, (a) Since Xi{k + 1) = Pxi[vi{k) — a{k)di{k)], it follows from the property of the 
projection error ei{k) in Eq. ^ that for any z E X, 

\\x^{k + 1) - zf < \\vi{k) - a{k)d,{k) - zf - \\e,{k)f. 

By expanding the term — a{k)di{k) — zW^, we obtain 

\\vi{k) - a{k)d,{k) - ;zf = \\vi{k) - + a^{k)\\di{k)\\'^ - 2a{k)di{ky{v^{k) - z). 

Since Vi{k) = X^jli (^ij{k)xj{k), using the convexity of the norm square function and the 
stochasticity of the weights aij{k), j = 1, . . . , m, it follows that 

Mk)~zr<Y^,,m^,{k)-zr. 

Combining the preceding relations, we obtain 

m 

\\x,{k+i)-zf < Y'^vm^jW-zf+a'mdim' 

-2a{k)di{ky{v,{k) - z) - \\e,{k)f. 

By summing the preceding relation over i = 1, . . . ,m, and using the doubly stochasticity 
of the weights, i.e.. 



m / m \ m 

2 



2=1 i = l j = l \i = l / j = l 

we obtain the desired result. 

(b) Since di{k) is a subgradient of fi{x) at x = Vi{k), we have 

diikYiv^ik) - z) > f^iviik)) - f,{z). 

Combining this with the inequality in part (a), using subgradient boundedness and 
dropping the nonpositive projection error term on the right handside, we obtain 

mm m 

J2 + 1) - < E \\x^{k) ~zr + a'{k)mL' - 2a{k) - f.{z)), 

1=1 1=1 i=l 
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proving the first claim. This relation implies that 

mm m 

W^Ak + 1) - zf < J2 -zf + a\k)mL^ - 2a{k) ^ {fMik)) - My{k))) 

-Mk) {f{y{k)) - f{z)) . ' ' (44) 

In view of the subgradient boundedness and the stochasticity of the weights, it follows 

m 

\Mvi{k)) - My{k))\ < L\\v,{k) - y{k)\\ < L a.,(fc)||a;,(A;) - y{k)\\, 
implying, by the doubly stochasticity of the weights, that 



m / m 



- Myik))\ <lY - = - 

i=l j=l \i=l J j=l 



By using this in relation (44), we see that for any z G X, and all i and k, 



Y \\xjik + 1) - zf < Y \\^j(k) - zf + a^{k)mL^ + 2a{k)L ^ \\xj{k) - y{k)\\ 

-Mk) {f{y{k)) - f{z)) . 



Our goal is to show that the agent disagreements \\xi{k) — Xj{k)\\ converge to zero. To 
measure the agent disagreements \\xi{k)—Xj{k)\\, we consider their average ^ J2]jLi ^j{k), 
and consider the disagreement of agent estimates with respect to this average. In par- 
ticular, we define 



^ m 

y{k) = —Y^jik) for all k. (45) 
^ i=i 

We have 

y{k+i) = -Y vm -'^Y.d.{k) + -Y, em- 

When the weights are doubly stochastic, since Viik) = J2T=i (^ij{k)xj{k), it follows that 



y{k + 1) = y{k) _ ^ V dm + - V em- (46) 

m < ' m < ' 



m — ' m 

i=\ 1=1 



Under our assumptions, the next lemma provides an upper bound on the agent 
disagreements, measured by < ||a;j(A;) — y{k)\\ > for all i, in terms of the subgradient 



bounds, projection errors and the disagreement metric p{k,s) defined in Eq. (14). 
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Lemma 6. Let Assumptions^ and^hold. Let {xi{k)} be the sequence generated by 
the algorithm and {y{k)} be defined in Eq. (46). Then, for all i and k > 2, 

upper bound on \\xi{k) —y{k)\\ is given by 



an 



k-2 



{k)-y{k)\\ < mp(A;-l,0)^||xj(0)||+mL^p(A;-l,r + l)a(r) + 2a(A;-l)L 



r=0 



k—2 m -. m 

+ Y,P{k-l,r + l)J2 l|e.(r)|| + \\e,{k - 1)|| + - \\e,{k 



r=0 



Proof. From Eq. (13), we have for all i and k > s, 

m k—1 m 



X. 



(k + l) = J2[^ik,s)]ijXj{s) - ^^[$(A;,r + l)],,a(r)rf,(r) -a(A;)rfi(A;) 

j=l r=s j=l 

k—1 m 

r=s j=l 

Similarly, using relation (46), we can write for y{k + 1) and for all k and s with k > s, 

^ k~l m / 1 \ ^ 1 ^ m 

y{k + 1) = --EE "(^)^^- w - ^ E ^^(^) + - E E w + - E ^.•(^) 



r=s j=l 



m 



i=l 



m 



r=s j=l 



m 



Therefore, since y{s) = ^ Yl^=i ^ji^)^ we have for s = 0, 



\x. 



m-ym < E 



[$(^-1,0)],,- 



m 



k—2 m ^ 

+ EE mk-hr + l)],,-- a{r)\\d,{r)\\ 

r=0 j=l 

+a{k - l)\\d.{k - 1)11 + E - 1)11 



m 



k—2 m 
r=0 j=l 



[$(A;-l,r + l)], 



m 



\ej{r) 



rn 

+ ||e,(A:-l)|| + -El|e.(fc-l)||. 



Using the metric p{k,s) = maxj jg^ — ior k > s > [cf. Eq. (14)], and 

the subgradient boundedness, we obtain for all i and k > 2, 



k-2 



{k)-y{k)\\ < mp{k-l,0)'^\\xj{0)\\+mLj2pik-^,r + l)a{r) + 2a{k-l)L 

j=l r=0 
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k—2 m ^ m 

+Y,pik-i,r+i)J2 11^. wii + - 1)11 + - E - 1)11' 

r=0 j=l j=l 

completing the proof. ■ 

In proving our convergence results, we will often use the following result on the 
infinite summability of products of positive scalar sequences with certain properties. 
This result was proven for geometric sequences in [19]. Here we extend it for general 
summable sequences. 

Lemma 7. Let {f3i} and {7^} be positive scalar sequences, such that Yl'ilol^i < ^'^'^ 
limfc^oo 7fc = 0. Then, 



k 

lim y~] (3k-ae = 0. 

fc— >CXD ' 

In addition, ifYl^=olk < 00, then 

00 k 

^^/^k-e'ye < 00. 
k=o e=o 

Proof. Let e > be arbitrary. Since 7^ — )■ 0, there is an index K such that 7^ < e for 
all k > K. For all k > K + 1, we have 



K k K k 

'^k-e- 



/3fc_£7£ = X] (^k-ele + ^ (3k-ele < max 7t ^ (3k-e + e (3k 



0<t<K 

=K+i e=o e=K+i 



Since Yl'ilo A < 00, there exists B > such that X]^=ii-+i t^k-e = X]£=o^ ^ f^e ^ B for 
all A; > + 1. Moreover, since /^^-^ ~ Yl^=k-K Pt-, it follows that for all A; > + 1, 



k 



£=0 f=k-K 

Therefore, using Yl'ilo f^i ^ obtain 

k 

lim sup ^ /3fc-£7£ < (^B. 
Since e is arbitrary, we conclude that limsupj!,^oQ Yl^=o f^k-He = 0, implying 

k 

lim ^ /3fc-£7£ = 0. 



£=0 



Suppose now Xlfc7fc < Then, for any integer M > 1, we have 

M / k \ M M-e M 



fc=0 \£=0 / £=0 i=0 £=0 
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implying that 



k=0 \£=0 



£=0 



4.2 Convergence Analysis when Xi = X for all i 

In this section, we study the case when agent constraint sets are the same. We study 
the asymptotic behavior of the agent estimates generated by the algorithm ([s]) using 
Assumption [5] on the stepsize sequence. 

The next assumption formalizes our condition on the constraint sets. 

Assumption 6. The constraint sets Xi are the same, i.e., Xi = X for a closed convex 
set X. 

We show first that under this assumption, we can provide an upper bound on the 
norm of the projection error ||ej(A;) || as a function of the stepsize a{k) for all i and k > 0. 

Lemma 8. Let Assumptions^ and^ hold. Let {ej(/c)} be the projection error defined 
by Then, for all i and k > 0, the ei{k) satisfy 

\\ei{k)\\ < 2La{k). 



Proof. Using the definition of projection error in Eq. (|8]), we have 

ei{k) = Xi{k + 1) - Vi{k) + a{k)di{k). 
Taking the norms of both sides and using subgradient boundedness, we obtain 

||ei(A;)|| < \\x^{k + 1) - v^{k)\\ + a{k)L. 

Since Vi{k) = J2]Li (^ij{k)xj{k), the weight vector ai{k) is stochastic, and Xj{k) G Xj = 
X (cf. Assumption |6]), it follows that Vi{k) E X for all i. Using the nonexpansive 
property of projection operation [cf. Eq. ([2])] in the preceding relation, we obtain 

||ei(A;)|| < \\vi{k) - a{k)di{k) - Vi{k)\\ + a{k)L < 2a{k)L, 

completing the proof. ■ 

This lemma shows that the projection errors are bounded by the scaled stepsize 
sequence under Assumption |6] Using this fact and an additional assumption on the 
stepsize sequence, we next show that the expected value of the sequences {||a;j(A;)— 
converge to zero for all i, thus establishing mean consensus among the agents in the limit. 
The proof relies on the bound on the expected disagreement metric p{k, s) established 
in Proposition |2j The mean consensus result also immediately implies that the agent 
estimates reach almost sure consensus along a particular subsequence. 
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Proposition 4. Let Assumptions\l\\^ 3' 13 on(i[^/ioW. Assume also that the stepsize 
sequence {a{k)} satisfies Assumption^. Let {xi{k)} be the sequence generated by the 
algorithm and {y{k)} be defined in Eq. (46). Then, for all i, we have 



lim E[\\xi{k) - y 
liminf ||xj(/c) — y{k)\\ = 



= 0, and 
with probability one. 



Proof. From Lemma |6| we have the following for all i and k >2, 

m k—2 

\\xi{k) -yik)\\ < mp{k - 1, 0) ^ ||xj(0)|| +mL^p{k - l,r + l)a{r) + 2a{k - 1)L 

j=l r=0 
k—2 m _ m 

+ J2p{k-l,r + l)J2 l|e.(r)|| + \\e,{k - 1)11 + - ^ W^^^^ ' 

lib 

r=0 j=l j=l 

Using the upper bound on the projection error from Lemmajsj ||ej(fc)|| < 2a{k)L for all 
i and k, this can be rewritten as 

m k—2 

\\xi{k) -y{k)\\ <mp{k-l,0)^\\xj{0)\\ + 3mL^ p{k - l,r + l)a{r) 

j=l r=0 

+ 6a{k-l)L. (47) 

Under Assumption [5] on the stepsize sequence, Proposition |2] implies the following bound 
for the disagreement metric p{k, s): for all > s > 0, 

E[p{k,s)]<P{s)e-''^% 

where /x is a positive scalar and /3(s) is an increasing sequence such that 

(3{s) < for all g > and all s > S{q), (48) 

for some integer S{q), i.e., for all q > 0, /3(s) is bounded by a polynomial s'^ for sufficiently 
large s (where the threshold on s, S{q), depends on q). Taking the expectation in Eq. 
(47) and using the preceding estimate on p{k, s), we obtain 

m k—2 

E[||xi(A;) -y(A;)||] <m/3(0)e-'^^^||x,(0)|| + 3mL ^ /3(r + l)e-^^'^^^a(r) 

j=l r=0 

+ 6a{k-l)L. 



We can bound /3(0) by /3(0) < S{1) by using Eq. ( [48j ) with q = 1 and the fact that (3 is 
an increasing sequence. Therefore, by taking the limit superior in the preceding relation 
and using a{k) — )■ as — j- oo, we have for all i, 



k-2 

limsupE[||a;i(A;) - y{k)\\] < 3mL ^ /3(r + l)e" 

k—>oo 



r=0 
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Finally, note that 



lim I3{k + l)a{k) < lim {k + l)a{k) = 0, 

fc— >oo k—>oo 



where the inequality holds by using Eq. (48) with q = 1 and the equality holds by 
Assumption [5] on the stepsize. Since we also have ^^lo^"^^ ^ Lemma [t] applies 
implying that 



fc-2 



lim y I3{r + l)e-^''^^^^^a{r) = 0. 

r=0 

Combining the preceding relations, we have 

limE[\\x,{k)-y{k)\\]=0. 



fc— >oo 



Using Fatou's Lemma (which applies since the random variables \\y{k) — Xi{k)\\ are 
nonnegative for all i and k), we obtain 



0< E 



liminf \\y{k) - Xi{k)\\ < hm mi E[\\y{k) - Xi{k)\\] < 0. 

. k—^00 J fc— ^-oo 



Thus, the nonnegative random variable liminffc_j,oo Wvik) — Xi{k)\\ has expectation 0, 
which implies that 



liminf \\y{k) — Xi{k)\\ = with probability one. 

fc— ^-oo 



The preceding proposition shows that the agent estimates reach a consensus in the 
expected sense. We next show that under Assumption [6} the agent estimates in fact 
converge to an almost sure consensus in the limit. We rely on the following standard 
convergence result for sequences of random variables, which is an immediate consequence 
of the supermartingale convergence theorem (see Bertsekas and Tsitsiklis |3j). 

Lemma 9. Consider a probability space {Q,F,P) and let {F{k)} be an increasing se- 
quence of a -fields contained in F. Let {V{k)} and {Z{k)} be sequences of nonnegative 
random variables (with finite expectation) adapted to {F{k)} that satisfy 

E[V{k + 1) I F{k)] < V{k) + Z{k), 

00 

J2E[Z{k)]< 00. 

k=l 

Then, V{k) converges with probability one, as k ^ 00. 

Proposition 5. Let Assumptions\l\\^ 3' <^nd\^hold. Assume also that the stepsize 
sequence {a{k)} satisfies Assumption^. Let {xi{k)} be the sequence generated by the 
algorithm and {y{k)} be defined in Eq. (46). Then, for all i, we have: 



(a) Yl'k'=2^i^)\\^i(^) ~ yi^)\\ < °° '"^^^^ probability one. 
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(b) limfc^oo — y{k)\\ = with probability one. 

Proof, (a) Using the upper bound on the projection error from Lemma [Sj ||ej(A;)|| < 
2a{k)L for all i and /c, in Lemma ^ we have for all i and k >2, 



m k—2 

\\xi{k) - y{k)\\ < mp{k - 1, 0) ^ \\xj{0)\\ + ?>mL^p{k - 1, r + l)a(r) + Qa{k - l)L. 

j=l r=0 

By multiplying this relation with a{k), we obtain 

m k—2 

a{k)\\x^{k) - y{k)\\ < ma{k)p{k - 1,0) ^ \\xj{0)\\ + 3mL^p{k - l,r + l)a{k)a{r) 

j=l r=0 

+ 6a{k)a{k -1)L. 
Taking the expectation and using the estimate from Proposition |2| i.e., 

E[p{k, s)] < /3(s)e-'^^^ for all A; > s > 0, 
where is a positive scalar and /3{s) is a increasing sequence such that 

(3{s) < for all g > and all s > S{q), (49) 

for some integer S{q), we have 

m 

E[a{k)\\x,{k)-y{k)\\] < ma(A;)/3(0)e-'^-^^||x,(0)|| 

k~2 

+3mL J2 + l)e-^^^-"-^a(A;)a(r) + 6a{k)a{k - 1)L. 

r=0 

Let ^(r) = P{r + l)a(r) for all r > 0. Using the relations a{k)C,{r) < a'^{k) + ^^(r) and 
2a{k)a{k — 1) < a'^{k) + a'^{k — 1) for any k and r, the preceding implies that 

m k—2 



E[a{k)\\x,{k) - y{k)\\] < ma(A:)/3(0)e-^^^^||a;j(0)|| +3mL^e-'^^^-'-2^2(^) 

j=l r=0 

k-2 

+3La\k) (m ^ e'^^'^^^^ + l) + 3a\k - 1)L. 

r=0 

Summing over k > 2, we obtain 

oo m CO 

J2E[a{k)\\x.{k)-y{k)\\] < m^||x,(0)||/3(0)^a(A;)e-^^ 

k=2 j=l k=2 

oo / k-2 

+3L^ (mY,e-^^^^^ + l^a\k) + a\k - 1) 

k=2 \ r=0 
oo k—2 



+3mL Yl e-^'^'^-'-^^^r) 



k=2 r=0 
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We next show that the right handside of the above inequahty is finite: Since hmfc_s>oo = 
(cf. Assumption 5), /3(0) is bounded, and Ylik^~^^ < Lemma [t] imphes that the 
first term is bounded. The second term is bounded since Ylik (^"^(M) < C)0 by Assumption 
5] and Lemma |4j Since ^(r) = (3{r + l)Q;(r), we have for some small e > and all r 
sufficiently large 

^2(^) = j^^i^r + l)a'(r) < (r + 1)2/V(r) < (r + if'^^. 



where the first inequality follows using the estimate in Eq. ( [49| with g = 1/3 and the 
second inequality follows from Assumption [sj This implies that Ylk^'^i^) ^ which 
combined with Lemma [7] implies that the third term is also bounded. Hence, we have 

00 

Y,E[amx^{k)-y{k)\\]<cx,. 

k=2 

By the monotone convergence theorem, this implies that 



E\J2c^myik)-x,ik)\\ 

and therefore 



k=2 



< 00, 



''^^a{k)\\y{k) ~ Xi(k)\\ < 00 with probability 1, 

k=2 

concluding the proof of this part. 

(b) Using the iterations ([T]) and (46), we obtain for all /c > 1 and i, 

y{k + l) -Xi{k + 1) = (^y{k) -J2(^^jik)xj{k)^ - a{k)(^-J2dj{k)-di{k)^ 



m 



+ {ij:^Ak)-em)- 



By the stochasticity of the weights aij{k) and the subgradient boundedness, this implies 
that 

m „ m 

\\y{k + 1) - x,{k + 1)11 < 5^ a,^{k)\\y{k) - x^{k)\\ + 2La{k) + - l|e,(fc)||. 

i=i ^ i=i 

Using the bound on the projection error from Lemma |8| we can simplify this relation as 

m 

\\y{k + 1) - Xi{k + 1)11 < ^a,,(fc)||i/(fc) - x,{k)\\ + 6La{k). 

i=i 

Taking the square of both sides and using the convexity of the squared-norm function 
II ■ II 2, this yields 



\yik+l)-Xiik+l)f < ^ai,-(A;)|||/(A;)-x,-(A;)f +12L«(A;) ^a,,(A;)|||/(A;)-x,(A;)||+36L 

i=i i=i 
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'a 



Summing over all i and using the doubly stochasticity of the weights aij{k), we have for 
all k>l, 

mm m 

Y^\\y(k+l)-Xi{k+l)f <Y,\\yik)-x.m\'+^'^La{k)Y^ 

i=l i=l i=l 

By part (a) of this lemma, we have YlT=i ~ < ^ with probability one. 

Since, we also have Ylk'^'^W ^ i'^^- Lemma |4]), Lemma [9] applies and implies that 
Slii \\y{^) ~ converges with probability one, as — )■ 00. 

By Proposition |4| we have 

liminf ||xj(A;) — y{k)\\ = with probability one. 

fc— ^-oo 

Since Yl^i WvW ~ converges with probability one, this implies that for all i, 

lim ||a;j(A;) — y{k)\\ = with probability one, 

fc— >oo 

completing the proof. ■ 

We next present our main convergence result under Assumption [5] on the stepsize 
and Assumption [6] on the constraint sets. 

Theorem 1. Let Assumptions [I| j^] anc? hold. Assume also that the stepsize 

sequence {a{k)} satisfies Xlfclo '^(^) ~ ^ ^'^^ Assumption^ Let {xi{k)} be the sequence 
generated by the algorithm Then, there exists an optimal solution x* G X* such 

that for all i 

lim Xi{k) = X* with probability one. 
Proof. From Lemma |5](b), we have for some z* E X* (i.e., f{z*) = /*), 

mm m 

^||x,(A; + l) < ^||a;j(A;) + a2(fc)mL2 + 2a(fc)L^||a;j(A;) -y(A;)|| 

-Mk) {f{y{k)) - r) , ' ' (50) 



[see Eq. (43)]. Rearranging the terms and summing these relations over k = 0, . . . , K, 
we obtain 

K mm 

2j2c^{k){f{y{k))-f*) < 5^||x,(0)-z*f -^||x,(ir + l)-^*f 

k=0 j=l j=l 

K Km 

+mL2 ^a2(A;) + 2L^a(A;) ^ ||a;j(fc) - ?/(A;)||. 

k=0 k=0 j=l 



By letting — )■ 00 in this relation and using YlT=o'^'^(^) ^ (^^- Lemma |4]) and 
^^o'-'^(^) ^j=i ll-^il^) ^ 2/(^)11 with probability one, we obtain 



K 



^^a;(/c) {f{y{k)) — /*) < 00 with probability one. 



A:=0 
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Since Xj(fc) e X for all i, we have y{k) G X [cf. Eq. (45)] and therefore f{y{k)) > f* for 
all k. Combined with the assumption YlT=o '^{^) = ^? the preceding relation implies 



]immifiy{k)) = r. 

fc— >oo 



(51) 



By dropping the nonnegative term 2a{k) {f{y{k)) — /*) in Eq. (50), we have 



j2\\xj{k+i) 



m 



|a;j(fc) 



a\k)mL'^ + 2a{k)Lj2\\^jik) - y{k)\\.{52) 



Since Yl'h=o^'^(^) ^ ^ ^^"^ SfcLo'^(^) Sj^i ll-^i(^) ~ 1/(^)11 < ^ with probability one, 
Lemma p| applies and implies that YlT=i ll^i(^) ~ is a convergent sequence with 



probability one for all z* G X*. By Lemma[5[b), we have limfc_>oo \\xiik)—yik)\\ = with 
probability one, therefore it also follows that the sequence \\y{k) — z*\\ is also convergent. 



Since y{k) is bounded, it must have a limit point. By Eq. (51) and the continuity of 
/ (due to convexity of / over M"), this implies that one of the limit points of {y{k)} 
must belong to X*; denote this limit point by x*. Since the sequence {|||/(A;) — is 
convergent, it follows that y{k) can have a unique limit point, i.e., \imk^ooy{k) = x* 
with probability one. This and limfc_!.oo ||a;j(fc) — y{k)\\ = with probability one imply 
that each of the sequences {xi{k)} converges to the same x* G X* with probability one. 



4.3 Convergence Analysis for Different Constraint Sets 

In this section, we provide our convergence analysis for the case when all the constraint 
sets Xi are different. We show that even when the constraint sets of the agents are 
different, the agent estimates converge almost surely to an optimal solution of problem 
^ under some conditions. In particular, we adopt the following assumption on the 
constraint sets. 

Assumption 7. For each i, the constraint set Xj is a convex and compact set. 

An important implication of the preceding assumption is that for each i, the subgra- 
dients of the function fi at all points a; G Xj are uniformly bounded, i.e., there exists 
some scalar L > such that for all i, 

\\d\\ < L for all d G dfi{x) and all x G Xj. 

Our first lemma shows that with different constraint sets and a stepsize that goes to 
zero, the projection error ei{k) converges to zero for all i along all sample paths. 



Lemma 10. Let Assumptions 
generated by the algorithm 
as k goes to infinity. 



3 and^hold. Let {xi{k)} and {ei{k)} be the sequences 
'8). Assume that the stepsize sequence satisfies a{k) — > 



(a) For any z G X , the scalar sequence X^Ili ll^i(^) " -^P ^■^ convergent. 



36 



(b) The projection errors ei{k) converge to zero as k ^ oo, i.e., 

lim ||ej(A;)|| = for all i. 



Proof, (a) Using subgradient boundedness and the relation |(ij(A;)'(t>j(A;)—z) I < ||(ij(A;) || ||t>j(A;)- 
z\\ in part (a) of Lemma Isl we obtain 



m 



J]||x,(A; + l)-2f <^\\x^{k)-zf + a^{k)mL^+2a{k)L^\\v^{k)-z\\-^\\e,{k)\\''. 

i=l i=l i=l i=l 

Since Vi{k) = Yl^=i'^iji^)^ji^)^ using doubly stochasticity of the weights, we have 
Yl^i ll^j(^) ~" ^11 — Si^Li ll^i(^) ^ -^11 5 which when combined with the preceding yields 
for any z ^ X and all A; > 0, 

mm mm 

\\xi{k+l)-z\\^ < \\xi{k)-z\\'^+a^{k)mL'^+2a{k)LY\\xiik)-z\\-Yl W^^i^^W'^- 

1=1 1=1 i=l 1=1 

(53) 

Since Xi{k) E Xi for all i and Xi is compact (cf. Assumption [?]), it follows that the 
sequence {xi{k)} is bounded for all and therefore the sequence il^j(^) ~ -^ll is 

bounded. Since a{k) — )■ as — )■ oo, by dropping the nonnegative term YllLi 



zr 



in Eq. (53), it follows that 

m m 

lim sup > ||xj(A; + 1) — z|p < liminfN ||a;i(A;) 

h^r^ fe— >00 ^ 

i=l 

(m 
a^{k)mL^ + 2a{k)L \\xi{k) 

m 

liminf > ||3;j(A;) — 



fc— ^OO . , . , 

1 = 1 1 = 1 



z 

k—^oo 

1=1 



i=l 

|2 



Since the sequence YlT=i ~ Al"^ is bounded, the preceding relation implies that the 

scalar sequence ll^j(^) ~ -^IP is convergent. 



(b) From Eq. (53), for any z G X, we have 



Y ^ \\^^{k)-z\\''-Y \\xi{k+l)-zf+a\k)mL'' + 2a{k)LY\Mk)--^ 

i=l i=l 1=1 i=l 

Taking the limit superior as A; — )■ oo, we obtain 

m 

2 



limsup> ||ej(A;)|p < lim |> ||xj(A;) — — > ||xi(A; + 1 
1=1 \i=l 1=1 

m 

+ lim (a^{k)mL'^ + 2a{k)L \\xi{k) - z\ 
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z\ 



i=l 



where the first term on the right handside is equal to zero by the convergence of the 
sequence Yl^=i W^ii.^) ~ A\'^i ^^^1 the second term is equal to zero by limfc_^oo Oi{k) = 
and the boundedness of the sequence Iki(^) ~ ^11 5 completing the proof. ■ 

The preceding lemma shows the interesting result that the projection errors ||ej(fc)|| 
converge to zero along all sample paths even when the agents have different constraint 
sets under the compactness conditions of Assumption [7j Similar to the case with Xi = X 
for all i, we next establish mean consensus among the agent estimates. The proof relies 
on the convergence of projection errors to zero and the bound on the disagreement 
metric p{k, s) from Proposition [s] Note that this result holds for all stepsizes a{k) with 
a{k) — )■ as — 7- 00. 

Proposition 6. Let Assumptions^ an(i[^ hold. Let {xi{k)} be the sequence gen- 
erated by the algorithm @-(^, and {y{k)} be defined in Eq. (46). Assume that the 



stepsize sequence satisfies a{k) as k goes to infinity. Then, for all i, we have 

lim E[\\x^{k) - y{k)\\] = 0, and 

liminf — y{k)\\ = with probability one. 

k—^oo 



Proof. From Lemma |6j we have 

m k—2 

\\xi{k) -y{k)\\ < mp{k - 1, 0) ^ ||xj(0)|| +mL^p{k - l,r + l)a{r) + 2a{k - 1)L 

j=l r=0 
k—2 m ^ m 

+ J]p(A; - l,r + 1) 5^ ||e,(r)|| + \\e.{k - 1)|| + - ^ W^^^^ '^^W 

r=0 j=l j=l 

Taking the expectation of both sides and using the estimate for the disagreement metric 
p(A;, s) from Proposition |3| i.e., for all > s > 0, 

E[p{k,s)] < Ke-^'^^~'\ 

for some scalars k, p > 0, we obtain 

m k—2 



E[\\xi{k) - y{k) II] < m/te-'^^'^-^) ^ \\xj{0) \\ + tuLk e"^^^-""^)a(r) + 2a{k - 1)L 

j=l r=0 
k—2 m ^ m 

+KY,e'^^'-^-'^ lle.WII + ||e,(A: - 1)11 + - ^ W^^^^ " 1)11- 



m 

r=0 j=l j=l 



By taking the limit superior in the preceding relation and using the facts that a{k) — )■ 0, 
and ||ej(A;)|| — )■ for alH as — ?■ 00 (cf. Lemma [ToJ^b)), we have for all 



fc-2 fc-2 m 

limsup^[||x,(A;) -?/(A;)||] < mLt^^e-^'^^-^-^^air) + k ^ e-^^'^-'^-^^ ^ 

r=0 r=0 j=l 



fc— >oo 
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Finally, since Yl'kLo^ '^^ < oo and both a{k) — )■ and ||ei(fc)|| — )■ for all i, by Lemma 
[7| we have 



fc-2 k-2 



lim y e-'^('^-'-2)a(r) = and lim e-^C^-'-^^ V ||e,(r)|| = 0. 

r=0 r=0 J=l 

Combining the preceding two relations, we have 

lim E[\\x^{k)-y{k)\\]=0. 

k—¥oo 

The second part of proposition follows using Fatou's Lemma and a similar argument 
used in the proof of Proposition |4j ■ 

The next proposition uses the compactness of the constraint sets to strengthen this 
result and establish almost sure consensus among the agent estimates. 

Proposition 7. Let Assumptions^ and[^ hold. Let {xi{k)} be the sequence gen- 
erated by the algorithm @-(^, and {y{k)} be defined in Eq. (46). Assume that the 
stepsize sequence satisfies a{k) — )• 0. Then, for all i, we have 

lim — y{k)\\ = with probability one. 

k—^oo 



Proof. Using the iterations ([T]) and (46), we obtain for all /c > 1 and i, 



y{k + 1) — Xi{k + 1) = (^y{k) — ''^^aij{k)xj{k)^ — a{k)(^ — dj{k) — di{k) 

^ m 

+ {-J2^^ik)-e.{k)). 

Using the doubly stochasticity of the weights aij{k) and the subgradient boundedness 
(which holds by Assumption [7| , this implies that 

mm m 

J2 \\y{k + 1) - Xi{k + 1)\\<J2 \\yW - Xi{k)\\ + 2Lma{k) + 2 J] ||e,(A;)||. (54) 

i=l i=i i=l 



Since a{k) — 0, it follows from Lemma [lO[b) that ||ej(A;)|| — )■ for all i. Eq. (54) then 
yields 



fe— >-oo . , A;— s>oo 

1=1 1=1 



lim sup > ||?/(A; + 1) — Xi(fc + 1) II < liminf > ||?/(A;) — a;i(A;)|| 

i=l 

m 

+ hm {2Lma{k) + 2'^\\ei{k)\\ 

°° i=l 
m 

liminf \\y{k) - Xi{k)\\. 



1=1 
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Using Xi{k) E Xi for all i and fc, it follows from Assumption [7] that the sequence {xi{k)} 
is bounded for all i. Therefore, the sequence {y{k)} [defined by y{k) = ^™ i a;j(/c), 
see Eq. (45)], and also the sequences \\y{k) — Xi{k)\\ are bounded. Combined with the 
preceding relation, this implies that the scalar sequence J21Li Wvi^) ~ is conver- 

gent. 

By Proposition [6| we have 

liminf ||xj(A;) — y{k)\\ = with probability one. 

k—^oo 

Since Yl^i WuW ~ converges, this implies that for all i, 

lim ||a;j(A;) — y{k)\\ = with probability one, 

completing the proof. ■ 

The next theorem states our main convergence result for agent estimates when the 
constraint sets are different under some assumptions on the stepsize rule. 

Theorem 2. Lei y4sswmj» tons and /ioW. Let {xi{k)} be the sequence generated 
by the algorithm Assume that the stepsize sequence satisfies J2k'^i^) ~ ^ ^''^^ 

^^q;^(A;) < oo. Then, there exists an optimal solution x* G X* such that for all i 

lim Xi{k) = X* with probability one. 
Proof. From Lemma |5](b), we have for some z* E X*, 

mm mm 

J2 \\x.{k+i)-z*r < J2 H{k)f-Mk) Y.{f.{v.m-M^*))- 

1=1 i=l 1=1 i=l 

(55) 

We show that the preceding implies that 

m 

\imMj2Mvi{k))<f{z*) = f*. (56) 

i=l 

Suppose to arrive at a contradiction that Mmmik^ooYl^i fii'^^iW) > /*• This implies 
that there exist some K and e > such that for all k > K, we have 



j2 f^iv,{k)) > r + e. 



1=1 



Summing the relation (55) over a window from K to N with N > K, we obtain 

m m N N 

\\xi{N + 1) - z*f < \\xi{K) - z*f + mL^ Y "'('^) " ^ a{k). 

1=1 i=l k=K k=K 

Letting iV — )• oo, and using a{k) = oo and J2k <^^(^) < oo, this yields a contradiction 
and establishes the relation in Eq. (56). 
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By Proposition [7} we have 

lim — y{k)\\ = with probabihty one. 



fc— ^-oo 



(57) 



Since Vi{k) = '^^=iCiij{k)xj{k), using the stochasticity of the weight vectors ai{k), this 
also imphes 



hm \\vi{k) — y{k)\\ < hm > aij{k)\\xj{k) — y{k)\\ =0 with probabihty one. (58) 



fe— >oo 



J = l 



Combining Eqs. (56) and (58), we obtain 



hminf f{y{k)) < f* with probabihty one. 

k—^oo 



(59) 



From Lemma 10 ^a), the sequence l|3;i(A;) 



*||} is convergent for ah z* e X*. 
Combined with Eq. (57), this imphes that the sequence {||y(fc) — ^*||} is convergent with 



probabihty one. Therefore, y{k) is bounded and it must have a hmit point. Moreover, 
since Xi{k) G Xj for aU > and Xi is a closed set, all limit points of the sequence 



{xi{k)} must lie in the set X^ for all i. In view of Eq. (57), this implies that all limit 



points of the sequence {y{k)} belong to the set X. Hence, from Eq. (59), we have 



lim inf f{y{k)) 

k—^oo 



f* with probability one. 



Using the continuity of / (due to convexity of / over M"), this implies that one of the 
limit points of {y{k)} must belong to X*; denote this limit point by x*. Since the 
sequence {||?/(fc) — is convergent, it follows that y{k) can have a unique limit point, 
i.e., limfc^oo2/(fc) = x* with probability one. This and limfc_j.oo ||a^i(^) — y{k)\\ = with 
probability one imply that each of the sequences {xi{k)} converges to the same x* G X* 
with probability one. ■ 



5 Conclusions 

We studied distributed algorithms for multi-agent optimization problems over randomly- 
varying network topologies. We adopted a state-dependent communication model, in 
which the availability of links in the network is probabilistic with the probability depen- 
dent on the agent states. This is a good model for a variety of applications in which the 
state represents the position of the agents (in sensing and communication settings), or 
the behefs of the agents (in social settings) and the distance of the agent states affects 
the communication and information exchange among the agents. 

We studied a projected multi-agent subgradient algorithm for this problem and pre- 
sented a convergence analysis for the agent estimates. The first step of our analysis es- 
tablishes convergence rate bounds for a disagreement metric among the agent estimates. 
This bound is time-nonhomogeneous in that it depends on the initial time. Despite this, 
under the assumption that the stepsize sequence decreases sufficiently fast, we proved 
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that agent estimates converge to an almost sure consensus and also to an optimal point 
of the global optimization problem under some assumptions on the constraint sets. 

The framework introduced in this paper suggests a number of interesting further re- 
search directions. One future direction is to extend the constrained optimization problem 
to include both local and global constraints. This can be done using primal algorithms 
that involve projections, or using primal-dual algorithms where dual variables are used 
to ensure feasibility with respect to global constraints. Another interesting direction 
is to consider different probabilistic models for state-dependent communication. Our 
current model assumes the probability of communication is a continuous function of the 
I2 norm of agent states. Considering other norms and discontinuous functions of agent 
states is an important extension which is relevant in a number of engineering and social 
settings. 
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